Method and system for processing an image

ABSTRACT

A method of processing an image is disclosed. The method comprises decomposing the image into a plurality of channels, each being characterized by a different depth-of-field, and accessing a computer readable medium storing an in-focus dictionary defined over a plurality of dictionary atoms, and an out-of-focus dictionary defined over a plurality of sets of dictionary atoms, each set corresponding to a different out-of-focus condition. The method also comprises computing one or more sparse representations of the decomposed image over the dictionaries.

RELATED APPLICATIONS

This application is a National Phase of PCT Patent Application No.PCT/IL2015/050587 having International filing date of Jun. 10, 2015,which claims the benefit of priority under 35 USC § 119(e) of U.S.Provisional Patent Application No. 62/010,000 filed on Jun. 10, 2014.The contents of the above applications are all incorporated by referenceas if fully set forth herein in their entirety.

The work leading to this disclosure has received funding from theEuropean Research Council under the European Union's Seventh FrameworkProgramme (FP7/2007-2013)/ERC grant agreement no. 335491.

FIELD AND BACKGROUND OF THE INVENTION

The present invention, in some embodiments thereof, relates to imagingand, more particularly, but not exclusively, to a method and a systemfor image processing.

Digital cameras are widely used due to high quality and low cost CMOStechnology and the increasing popularity of social network. The demandfor high resolution and quality cameras, specifically for smart phones,led to a competitive market that constantly tries to create a bettercamera.

Digital image quality is determined by the imaging system properties andfocal plane array sensor. With the increase in pixel number and density,the imaging system resolution is bound now mostly by optical systemlimitation. The limited volume in smart phones makes it very difficultto improve the image quality by optical solutions and therefore most ofthe advancements in recent years were software related.

Ref [1] discloses a binary symmetrical phase mask that allows increasingthe camera's Depth of Field (DOF) for different uses such as barcodereading, face detection as well as other computer vision relatedapplications. Ref. [2] discloses a RGB phase mask, whereby one getsdifferent responses for the R, G and B channels, resulting insimultaneous capturing of three images, each with a differentout-of-focus characteristics, but with the same magnification and inperfect registration. Different regions in the object space may exhibitgood gray level features in one channel and poorer gray images in theother two channels. The three channels (RGB) jointly analyzed enable theextended DOF system response.

Additional background art includes U.S. Pat. No. 8,682,066, the contentsof which are hereby incorporated by reference.

SUMMARY OF THE INVENTION

According to an aspect of some embodiments of the present inventionthere is provided a method of processing an image. The method comprises:decomposing the image into a plurality of channels. In some embodiments,each channel is characterized by a different depth-of-field. The methodfurther comprises accessing a computer readable medium storing anin-focus dictionary defined over a plurality of dictionary atoms, and anout-of-focus dictionary defined over a plurality of sets of dictionaryatoms, each set corresponding to a different out-of-focus condition; andcomputing at least one sparse representation of the decomposed imageover the dictionaries.

According to some embodiments of the invention the decomposition is doneoptically at the time of image capture.

According to some embodiments of the invention the decomposition is doneby digital image processing.

According to an aspect of some embodiments of the present inventionthere is provided a method of imaging. The method comprises: capturingan image of a scene by an imaging device having an optical mask selectedto optically decompose the image into a plurality of channels. In someembodiments of the present invention each channels is characterized bydifferent depth-dependence of a spatial frequency response of theimaging device. The method further comprises accessing a computerreadable medium storing an in-focus dictionary defined over a pluralityof dictionary atoms, and an out-of-focus dictionary defined over aplurality of sets of dictionary atoms, each set corresponding to adifferent out-of-focus condition; and computing at least one sparserepresentation of the decomposed image over the dictionaries.

According to an aspect of some embodiments of the present inventionthere is provided an imaging system. The system comprises: an imagingdevice having an optical mask and being configured for capturing animage of a scene, wherein the optical mask is selected to opticallydecompose the image into a plurality of channels. In some embodiments ofthe present invention each channels is characterized by differentdepth-dependence of a spatial frequency response of the imaging device.The system optionally and preferably comprises a computer readablemedium storing an in-focus dictionary defined over a plurality ofdictionary atoms, and an out-of-focus dictionary defined over aplurality of sets of dictionary atoms, each set corresponding to adifferent out-of-focus condition; and a data processor configured foraccessing the computer readable medium and computing at least one sparserepresentation of the decomposed image over the dictionaries.

According to an aspect of some embodiments of the present inventionthere is provided a portable device, comprising the imaging system.

According to some embodiments of the present invention the portable isselected from the group consisting of a cellular phone, a smartphone, atablet device, a mobile digital camera, a wearable camera, a personalcomputer, a laptop, a portable media player, a portable gaming device, aportable digital assistant device, and a portable navigation device.

According to an aspect of some embodiments of the present inventionthere is provided a computer software product. The computer softwareproduct comprises a computer-readable medium, optionally and preferablya non-transitory computer-readable medium, in which an in-focusdictionary, an out-of-focus dictionary and program instructions arestored, wherein the in-focus dictionary is defined over a plurality ofdictionary atoms, and the out-of-focus dictionary defined over aplurality of sets of dictionary atoms, each set corresponding to adifferent out-of-focus condition, and wherein the instructions, whenread by a data processor, cause the data processor to receive an image,to decompose the image into a plurality of channels, to access thein-focus dictionary and the out-of-focus dictionary, and to compute atleast one sparse representation of the decomposed image over thedictionaries.

According to some embodiments of the present invention the plurality ofchannels is a plurality of color channels.

According to some embodiments of the present invention the computationof the sparse representation is executed without iteration.

According to some embodiments of the present invention the image isdivided into a plurality of patches, and the computation of the sparserepresentation is performed by expressing each patch as a combination ofatoms of a sub-dictionary of the out-of-focus dictionary.

According to some embodiments of the invention each patch of at least afew (e.g., at least 50% or at least 60% or at least 70%) of the patchesoverlaps with at least one adjacent patch.

According to some embodiments of the invention for each of at least afew pairs of adjacent patches, the overlap equals at least 50% or atleast 60% or at least 70% or at least 80% of an area of each patch ofthe pair.

According to some embodiments of the present invention the out-of-focusdictionary comprises a plurality of sub-dictionaries, each beingcharacterized by a defocus parameter, and wherein differentsub-dictionaries correspond to different values of the defocusparameter.

According to some embodiments of the invention each of the plurality ofsub-dictionaries is obtainable from the in-focus dictionary by an innerproduct of the in-focus dictionary by a kernel function characterized bya respective defocus parameter.

According to some embodiments of the invention the out-of-focusdictionary comprises at least three sub-dictionaries, more preferably atleast four sub-dictionaries, more preferably at least fivesub-dictionaries, more preferably at least six sub-dictionaries, morepreferably at least seven sub-dictionaries.

According to some embodiments of the invention at least P % of atoms ofthe in-focus dictionary are characterized by a spatial frequency whichis at most a predetermined cutoff frequency, the predetermined cutofffrequency corresponding to at most T transitions between dark and brightpicture-elements along any straight line across a respective atom,wherein P is at least 50 or at least 60 or at least 70 or at least 80 orat least 90, wherein T equals └0.5 ┘ or └0.4 ┘ or └0.3 ┘ or └0.2 ┘, andwherein L is a width of the atom.

Unless otherwise defined, all technical and/or scientific terms usedherein have the same meaning as commonly understood by one of ordinaryskill in the art to which the invention pertains. Although methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing of embodiments of the invention, exemplarymethods and/or materials are described below. In case of conflict, thepatent specification, including definitions, will control. In addition,the materials, methods, and examples are illustrative only and are notintended to be necessarily limiting.

Implementation of the method and/or system of embodiments of theinvention can involve performing or completing selected tasks manually,automatically, or a combination thereof. Moreover, according to actualinstrumentation and equipment of embodiments of the method and/or systemof the invention, several selected tasks could be implemented byhardware, by software or by firmware or by a combination thereof usingan operating system.

For example, hardware for performing selected tasks according toembodiments of the invention could be implemented as a chip or acircuit. As software, selected tasks according to embodiments of theinvention could be implemented as a plurality of software instructionsbeing executed by a computer using any suitable operating system. In anexemplary embodiment of the invention, one or more tasks according toexemplary embodiments of method and/or system as described herein areperformed by a data processor, such as a computing platform forexecuting a plurality of instructions.

Optionally, the data processor includes a volatile memory for storinginstructions and/or data and/or a non-volatile storage, for example, amagnetic hard-disk and/or removable media, for storing instructionsand/or data. Optionally, a network connection is provided as well. Adisplay and/or a user input device such as a keyboard or mouse areoptionally provided as well.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

Some embodiments of the invention are herein described, by way ofexample only, with reference to the accompanying drawings and images.With specific reference now to the drawings in detail, it is stressedthat the particulars shown are by way of example and for purposes ofillustrative discussion of embodiments of the invention. In this regard,the description taken with the drawings makes apparent to those skilledin the art how embodiments of the invention may be practiced.

In the drawings:

FIG. 1 is a schematic illustration of a system of coordinates useful fordescribing a thin-lens imaging system, according to some embodiments ofthe present invention;

FIG. 2 shows Modulation Transfer Function (MTF) as calculated for acircular aperture for different value of a defocus parameter ψ accordingto some embodiments of the present invention;

FIGS. 3A-D show comparison between MTF response with and without an RGBmask, according to some embodiments of the present invention;

FIGS. 4A and 4B show comparison between MTF responses of a singlefrequency as a function of a defocus parameter ψ in case of clearaperture (FIG. 4A) and aperture with an RGB mask (FIG. 4B), obtainedaccording to some embodiments of the present invention;

FIGS. 5A-R are Lena images obtained according to some embodiments of thepresent invention with a mask-equipped pupil (FIGS. 5D-F, 5J-L and5P-R), as well as by a clear aperture system (FIGS. 5A-C, 5G-I and5M-O), at a red channel (left column), a green channel (middle column),and blue channel (right column), for ψ=0 (FIGS. 5A-F), ψ=3 (FIGS. 5G-L)and ψ=6 (FIGS. 5M-R);

FIGS. 6A-D show randomly selected dictionaries (FIGS. 6A and 6B) and lowspatial frequency dictionaries (FIGS. 6C and 6D), before (FIGS. 6A and6C) and after (FIGS. 6B and 6D) imaging, according to some embodimentsof the present invention;

FIG. 7 is a schematic illustration of a procedure according toembodiments of the present invention in which a stack blurred dictionaryis constructed using an original dictionary and a plurality of differentblurring kernels corresponding to different defocus parameters;

FIGS. 8A-H show comparisons between several image processing techniquesfor eight different images, as obtained according to some embodiments ofthe present invention;

FIGS. 9A-F show example of image blurring and restoration, according tosome embodiments of the present invention, where FIG. 9A shows the inputimage, FIGS. 9B and 9C show images corresponding to out-of-focus (Ψ=6)with clear aperture (FIG. 9B) and with phase mask (FIG. 9C), FIGS. 9Dand 9E are results of de-blurring, respectively applied to FIGS. 9B and9C using a process found in [8], and FIG. 9F is a result of de-blurringapplied to FIG. 9C using a multi-focus dictionary (MFD);

FIGS. 10A-C exemplify a scene consisting of a finite number of flatobjects, where FIG. 10A shows objects located at different distance fromthe camera, FIG. 10B shows the input image, and FIG. 10C shows acorresponding depth map;

FIGS. 11A-C exemplify a restoration technique to a scene consisting of afinite number of flat objects without decomposing the image, where FIG.11A shows a defocused image, FIG. 11B shows the restored image, and FIG.11C shows a depth map according to the most used sub-dictionary in eacharea;

FIGS. 12A-C exemplify a restoration technique to a scene consisting of afinite number of flat objects with image decomposition, where FIG. 12Ashows a defocused image, FIG. 12B shows the restored image, and FIG. 12Cshows a depth map according to the most used sub-dictionary in eacharea;

FIGS. 13A-D show experimental results comparing images taken without aphase mask and without post processing (FIGS. 13A and 13C), with imagescaptured and processed according to some embodiments of the presentinvention (FIGS. 13B and 13D);

FIG. 14 is a flowchart diagram describing a method for processing animage, according to some embodiments of the present invention;

FIG. 15 is a schematic illustration of an imaging system, according tosome embodiments of the present invention; and

FIG. 16 is a schematic illustration of a phase mask, according to someembodiments of the present invention.

DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION

The present invention, in some embodiments thereof, relates to imagingand, more particularly, but not exclusively, to a method and a systemfor image processing.

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not necessarily limited in itsapplication to the details of construction and the arrangement of thecomponents and/or methods set forth in the following description and/orillustrated in the drawings and/or the Examples. The invention iscapable of other embodiments or of being practiced or carried out invarious ways.

Sparse representation of natural image signals is a powerful tool forimage representation, compression and restoration. By using an overcomplete dictionary of patterns, small patches from an image arepresented in the present examples as a linear combination of a few atoms(samples) from the dictionary. Known techniques assume that the blurringkernel is known [3-5] or use an iterative computing process to evaluatesuch kernel [6-9]. The present inventors found that the differencesbetween the responses of the RGB channels, e.g., when the imaging systemincludes a phase mask, can be used to restore the blurred image in asingle run without prior knowledge of the blurring kernel.

Integration between optics and image processing has been presented inmany Computational photography related work. Light field cameras [10]presented some abilities concerning focus manipulation and improvingDOF. Nevertheless, the present inventors found that light field camerassuffer from low resolution, noise and require a unique design whichmakes it harder to integrate them with existing systems, such as, butnot limited to, smartphones, laptop cameras, etc. Ref. [11] describes acoded aperture with a conventional camera to achieve similar results. Adifferent approach [12] utilize lens with deliberately high chromaticaberration to achieve color separation. The present inventors found thatthis approach requires special lens design for each system.

While conceiving the present invention it has been hypothesized andwhile reducing the present invention to practice it has been realizedthat representation of images using an out-of-focus dictionary can beimproved when the images are decomposed into channels, and thedictionary comprises a plurality of sets of dictionary atoms, where eachset corresponding to a different defocus condition. It was found by thepresent inventors that such a combination, between image decomposinginto a plurality of channels and out-of-focus dictionary with aplurality of sets of atoms, facilitates more accurate representationbecause it allows the representation to weigh the dictionary atomsaccording to the out-of-focus condition of each image region.

FIG. 14 is a flowchart diagram describing a method for processing animage, according to some embodiments of the present invention.Preferably, but not necessarily the method also captures the image.

The method can be embodied in many forms. For example, it can beembodied in on a tangible medium such as a computer for performing themethod operations. It can be embodied on a computer readable medium,comprising computer readable instructions for carrying out the methodoperations. It can also be embodied in an electronic device havingdigital computer capabilities arranged to run the computer program onthe tangible medium or execute the instruction on a computer readablemedium. A representative example of such an electronic device is dataprocessor of a mobile device, such as, but not limited to, a smartphoneor a tablet device.

Computer programs implementing the method according to some embodimentsof this invention can commonly be distributed to users on a distributionmedium such as, but not limited to, CD-ROM, flash memory devices, flashdrives, or, in some embodiments, drives accessible by means of networkcommunication, over the internet (e.g., within a cloud environment), orover a cellular network. From the distribution medium, the computerprograms can be copied to a hard disk or a similar intermediate storagemedium. The computer programs can be run by loading the computerinstructions either from their distribution medium or their intermediatestorage medium into the execution memory of the computer, configuringthe computer to act in accordance with the method of this invention.Computer programs implementing the method according to some embodimentsof this invention can also be executed by one or more data processorsthat belong to a cloud computing environment. All these operations arewell-known to those skilled in the art of computer systems. Data usedand/or provided by the method of the present embodiments can betransmitted by means of network communication, over the internet, over acellular network or over any type of network, suitable for datatransmission.

It is to be understood that, unless otherwise defined, the operationsdescribed hereinbelow can be executed either contemporaneously orsequentially in many combinations or orders of execution. Specifically,the ordering of the flowchart diagrams is not to be considered aslimiting. For example, two or more operations, appearing in thefollowing description or in the flowchart diagrams in a particularorder, can be executed in a different order (e.g., a reverse order) orsubstantially contemporaneously. Additionally, several operationsdescribed below are optional and may not be executed.

The method begins at 10 and optionally and preferably continues to 11 atwhich an image of a scene is acquired. The image can be captured by themethod using an imaging device, or received by the method from anexternal source, such as a computer readable medium storing a previouslycaptured image. The method preferably continues to 12 at which the imageis decomposed into a plurality of channels. The decomposing 12 can bedone optically, at the time of image capture, for example, by allowinglight from the scene to pass an optical mask that decomposes the lightinto a plurality of channels. Alternatively, the decomposing 12 can bedone by digital image processing, for example, while decomposing thedata stored in each picture-element (e.g., an image pixel) into aplurality of channels.

Each of the channels is characterized by a different range of effectivedepth-of-field (DOF). The DOF is typically parameterized using aparameter known as the defocus parameter Ψ. In typical imaging systems,the defocus parameter is, in absolute value, within the range 0 to 6radians, but other ranges are also envisioned. In some embodiments ofthe present invention each of the channels is characterized by adifferent depth-dependence of a spatial frequency response of theimaging device used for captured the image. The spatial frequencyresponse can be expressed, for example, as an Optical Transfer Function(OTF).

In various exemplary embodiments of the invention the channels aredefined according to the wavelengths of the light arriving from thescene. In these embodiments, each channel corresponds to a differentwavelength range of the light. As will be appreciated by one ofordinarily skilled in the art, different wavelength ranges correspond todifferent depth-of-field ranges and to different depth-dependence of thespatial frequency response. A representative example of a set ofchannels suitable for the present embodiments is a red channel,corresponding to red light (e.g., light having a spectrum having an apexat a wavelength of about 620-680 nm), a green channel, corresponding togreen light (spectrum having an apex at a wavelength of from about 520to about 580 nm), and a blue channel, corresponding to blue light(spectrum having an apex at a wavelength of from about 420 to about 500nm). Such a set of channels is referred to herein collectively as RGBchannels.

When the decomposing is executed optically, a phase mask, such as, butnot limited to, an RGB phase mask is optionally and preferably employed.The mask is preferably selected for optically delivering differentexhibited phase shifts for different wavelength components of the light.For example, the mask can generate phase-shifts for red light, for greenlight and for blue light, wherein each one of these phase-shifts isdifferent from the other two phase-shifts by an amount which is not 2nπ,where n is an integer. As a representative example, that is not to beconsidered as limiting, the first phase-shift can be about π radians(modulo 2π) the second phase-shift can be about π/2 radians (modulo 2π)and the third phase-shift can be close to zero, e.g., less than 0.1radians (modulo 2π).

For example, the phase mask can have one or more concentric rings thatmay form a grove and/or relief pattern on a transparent mask substrate.Each ring preferably exhibits a phase-shift that is different to thephase-shift of the remaining mask regions. The mask can be a binaryamplitude phase mask, but non-binary amplitude phase masks are alsocontemplated.

Optionally, but not necessarily, the mask includes a single ring thatextends between 0.1 and 0.7 of the radius of the mask. The mask can beplaced in the optical train of the imaging device, for example, in frontof the lens. The mask is optionally and preferably thin, for example,from about 1 μm to about 20 μm, or from about 2 μm to about 10 μm, orfrom about 2 μm to about 8 μm. The diameter of the mask depends on thediameter of the lens of the imaging device, and can any diameter be fromabout 1 mm to about 100 mm. A representative example of a mask suitablefor the present embodiments is described in U.S. Pat. No. 8,682,066 andRef. [2]. The mask can be supported by an optically transparentsupporting member, e.g., a glass plate, or be positioned on top of anexisting optical element, such as, but not limited to, a lens, e.g., alens of the imaging system.

The method continues to 13 at which a computer readable medium storingone or more dictionaries is accessed. In various exemplary embodimentsof the invention the computer readable medium stores an in-focusdictionary D and an out-of-focus dictionary D_(b). The in-focusdictionary D is defined over a set {s} of dictionary atoms, and theout-of-focus dictionary D_(b) is defined over a plurality of sets{b}_(j) of dictionary atoms, wherein each set of D_(b) corresponds to adifferent out-of-focus condition. For example, each set of atoms inD_(b) can correspond to a different value of the defocus parameter ψ.

Each set of atoms in D_(b) is referred to herein as a sub-dictionary ofD_(b). In some embodiments D_(b) comprises at least threesub-dictionaries (respectively corresponding to three differentout-of-focus conditions, e.g., three different values of ψ), in someembodiments D_(b) comprises at least four sub-dictionaries (respectivelycorresponding to four different out-of-focus conditions, e.g., fourdifferent values of ψ), in some embodiments D_(b) comprises at leastfive sub-dictionaries (respectively corresponding to five differentout-of-focus conditions, e.g., five different values of ψ), in someembodiments D_(b) comprises at least six sub-dictionaries (respectivelycorresponding to six different out-of-focus conditions, e.g., sixdifferent values of ψ), in some embodiments D_(b) comprises at leastseven sub-dictionaries (respectively corresponding to seven differentout-of-focus conditions, e.g., seven different values of ψ).

The dictionaries can form a complete basis or, more preferably, they canbe redundant dictionaries (also referred to in the literature asovercomplete dictionaries). A redundant dictionary is a dictionary thatincludes more atoms than the minimal number of base atoms required torepresent the image. A sparse representation typically includes aportion (e.g., less than 80% or less than 60% or less than 40% or lessthan 20% or less than 10% or less than 5%) of the atoms from theavailable atoms in the dictionary.

The in-focus dictionary D can be prepared, according to some embodimentsof the present invention, from a reference image which is preferably anatural image. The natural image can be captured by any imaging system.In experiments performed by the present inventors, the publiclyavailable “Lena” image has been used as a reference image. A largenumber of candidate atoms (e.g., at least 10,000 candidate atoms or atleast 20,000 candidate atoms or at least 40,000 candidate atoms or atleast 80,000 candidate atoms, e.g., 100,000 candidate atoms or more) areselected from the image. The candidate atoms can be patches of adjacentpicture-elements (e.g., patches of pixels). For example, each candidateatom can be a patch having the shape of a square of N×Npicture-elements, where N can be from about 4 to about 30, or from about4 to about 16, e.g., N=8. From the collection of candidate atoms, theatoms that constitute the dictionary can be thereafter selected. Theselection of atoms from the collection of candidate atoms is optionallyand preferably according to the similarity of each candidate atom withother candidate atoms in the collection, more preferably with all othercandidate atoms in the collection. The similarity can be determined bycalculating the inner product between the candidate atoms, whereinhigher inner product corresponds to higher similarity. In variousexemplary embodiments of the invention the in-focus dictionary includesN_(D) atoms that are most similar to all other candidate atoms in thecollection. The value of N_(D) is smaller than the number of candidateatoms in the collection. Typically, N_(D) is less than 1000 or less than800 or less than 600.

According to some embodiments of the invention at least a portion (e.g.,at least 50% or at least 60% or at least 70% or at least 80% or at least90%) of the atoms of the in-focus dictionary D are characterized by lowspatial frequency. It was found by the present inventors that use of lowspatial frequencies patches for D, increases the likelihood ofgenerating an accurate representation of the image.

As used herein, “low spatial frequency” means a spatial frequency whichis at most a predetermined cutoff frequency, the predetermined cutofffrequency corresponding to at most T transitions between dark and brightpicture-elements along any straight line across a respective atom, whereT can be └0.5┘ or └0.4┘ or └0.3┘ or └0.2┘, and where L is the width ofthe atom, and └.┘ denotes the Floor function.

Herein, the width of an atom or an image patch is measured along thesmallest dimension of the atom or image patch. Typically, the width isexpress in units of picture-elements (e.g., pixels).

The dictionaries D_(b) and D are preferably related via a set of kernelfunctions that represent image blurring resulting from the out-of-focusconditions. According to some embodiments of the invention, eachsub-dictionary of D_(b) is obtainable from the in-focus dictionary D(once constructed, e.g., as explained above) by an inner product of D bya respective kernel function characterized by a respective defocusparameter. This embodiment is illustrated in FIG. 7.

Representative examples of in-focus dictionaries suitable for thepresent embodiments are provided in FIGS. 6A and 6C. Representativeexamples of a set of atoms of an out-of-focus dictionary suitable forthe present embodiments are provided in FIGS. 6B and 6D.

The method continues to 14 at which a sparse representation of thedecomposed image over the dictionaries is computed. This can be done,for example, by dividing the decomposed image into a plurality ofpatches, and calculating for each atom b_(i) of D_(b) a coefficientα_(i) that estimates the contribution of the respective atom to an imagepatch. Specifically, a positive coefficient indicates that therespective atom is a component in the image, and a non-positivecoefficient (or a coefficient below a certain threshold) indicates thatthe image is devoid of the respective dictionary atom. In someembodiments of the present invention, the coefficients are calculatedunder a non-negativity constraint. This can be done, for example, byreplacing all negative coefficients by zeros. In some embodiments of thepresent invention the sparse representation is computed withoutiteration.

Many techniques for calculating such coefficients can be employed.Generally, these techniques include, but are not limited to, a pursuitalgorithm, e.g., Orthogonal Matching Pursuit, Matching Pursuit, BasisPursuit, Order Recursive Matching Pursuit, Focal Underdetermined SystemSolver, or any combination or variation thereof.

Each patch of at least a few (e.g., 50% or 60% or 70% or 80% or 90% orall) of the patches of the decomposed image preferably overlaps with atleast one adjacent patch. The overlap can equals at least 50% or atleast 60% or at least 70% or at least 80% of the area of each patch ofthe pair of overlapping patches.

The technique that calculates the coefficients (e.g., the pursuitalgorithm) selects from D_(b) the atom that best match the respectivepatch of the input image. Since D_(b) includes several sub-dictionaries,it varies significantly for different kernels. A patch of the decomposedinput image also exhibits different response for each out-of-focuscondition (e.g., for each wavelength range or color). The responses ofthe imaging system are therefore distinguishable among different kernelsso that the likelihood that the patch of the input image is inaccordance with atoms from D_(b) that experience the same out-of-focuscondition is high. Thus, the technique of the present embodimentsimproves the likelihood that the correct atoms are assigned with highercoefficients.

Once the coefficients are calculated, the image can be further processedusing the computed coefficients and the atoms of the in-focus dictionaryD. Typically, this process results in a sharpened image, because theatoms of D represent sharp patches. In other words, a sparserepresentation of the decomposed image using the calculated coefficientsα_(i) and the atoms s_(i) of D substantially inverts the effect of thekernels used to construct D_(b) from D, thus restoring a sharpenedimage.

Following the calculation of the sparse representation coefficientsα_(i), the value of the defocus parameter ψ is generally known for allthe patches of the input image that form the sparse representation. Thisvalue of ψ is typically the value that is associated with the kernelfunction from which the sub-dictionary that contains the atom that ismultiplied by the highest coefficient.

The distribution of ψ values among the patches can be used in more thanone way.

In some embodiments, a depth map of the scene is constructed, asindicated at 15. This can be done based on the sparse representation,or, more specifically, using the distribution of ψ values among thepatches. Preferably, the depth map is constructed based on a singleimage frame. The depth map describes relative depths of differentobjects in the scene. The relative depths can be expressed in the depthmap in units of normalized length, or, more preferably in terms of thedefocus parameter ψ. For example, the depth map can assign an estimatedvalue of ψ for each object in the scene. Since different values for ψcorrespond to different ranges between the object and the imagingdevice, the estimated values of ψ can be transformed into estimatedranges. Since the depth map correlates with range data, the method canuse the depth map to generate a three-dimensional (3D) representation(e.g., 3D image) of the scene, as indicated at to 16.

In some embodiments of the present invention the method continues to 17at which the image is refocused.

As used herein, “refocusing” means processing an input image which isobtained with an imaging device to form a reconstructed image (referredto herein as refocused image) which is focused on a predetermined planeother than the plane on which the imaging device was focused at the timeof image captured.

In the refocused image the sharpness of the objects at the predeterminedplane is typically higher than the sharpness of objects at other planes.

The refocusing operation is optionally and preferably executed using thesparse representation, or, more specifically, using the distribution ofψ values among the patches. For example, once an image region has beenidentified to be associated with a certain ψ value, the region can besharpened by using the sharp dictionary D with the same set of αcoefficients.

The method optionally continues to 18 at which the processed image isdisplayed on a display device. The processed image can also be recordedon a non-volatile computer-readable medium. The processed image can bethe sharpened image, the refocused image, the 3D image or any otherimage obtained using one or more of the dictionaries D and D_(b).

The method ends at 19.

FIG. 15 is a schematic illustration of an imaging system 30, accordingto some embodiments of the present invention. System 30 comprises animaging device 32 for capturing an image of a scene 38. Imaging device32 preferably comprises a device body 40, a lens 36 and an optical mask34 selected to optically decompose the image or the light from scene 38into a plurality of channel, as further detailed hereinabove. Mask 34 ispreferably mounted on lens 36. A representative example of mask 34 inembodiments in which the mask includes a single ring forming a reliefpattern is schematically illustrated in FIG. 16.

System 30 additionally comprises a computer readable medium 42 forstoring the in-focus dictionary D and out-of-focus dictionary D_(b), anda data processor 44 for accessing medium 42 and computing a sparserepresentation of the decomposed image over the dictionaries, as furtherdetailed hereinabove. Data processor can be configured for executing anyof the image processing operations described above with respect tomethod 10. Preferably, but not necessarily, data processor comprises adedicated circuit, for example, an application-specific integratedcircuit (ASIC) configured for executing these operations. Alsocontemplated is the use of a field-programmable gate array (FPGA) forperforming at least a few of the image processing operations.

Each of medium 42 and/or processor 44 can be separated from imagingdevice 32 or they can be incorporated in body 40. In some embodiments ofthe present invention system 30 comprises a display device 46 thatreceives the processed image generated by data processor 44 and displaysit.

System 30 can be incorporated in a portable device selected from thegroup consisting of a cellular phone, a smartphone, a tablet device, amobile digital camera, a wearable camera, a portable media player, apersonal computer, a laptop, a portable gaming device, a portabledigital assistant device, and a portable navigation device.

The system and method of the present embodiments enjoy many advantagesnot possessed by conventional systems.

One advantage of the system and method of the present embodiments isthat the technique can be adapted for lens systems that are fullycorrected. This is because the processing technique of the presentembodiments allows restoration after image capture. This is particularlyadvantageous when the system is incorporated in a mobile device such asa smartphone or a tablet, since short optical trains typically are notassociated with perfectly corrected lenses. This is also advantageousfrom the standpoint of cost since lens systems with shorter opticaltrains are less expensive.

The system and method of the present embodiments can also be implementedin low-light conditions. Conventional systems for capturing images underlow-light conditions suffer from the limitation of reduced DOF. Theprocessing technique of the present embodiments at least partiallyrestores the loss of DOF.

As used herein the term “about” refers to ±10%.

The word “exemplary” is used herein to mean “serving as an example,instance or illustration.” Any embodiment described as “exemplary” isnot necessarily to be construed as preferred or advantageous over otherembodiments and/or to exclude the incorporation of features from otherembodiments.

The word “optionally” is used herein to mean “is provided in someembodiments and not provided in other embodiments.” Any particularembodiment of the invention may include a plurality of “optional”features unless such features conflict.

The terms “comprises”, “comprising”, “includes”, “including”, “having”and their conjugates mean “including but not limited to”.

The term “consisting of” means “including and limited to”.

The term “consisting essentially of” means that the composition, methodor structure may include additional ingredients, steps and/or parts, butonly if the additional ingredients, steps and/or parts do not materiallyalter the basic and novel characteristics of the claimed composition,method or structure.

As used herein, the singular form “a”, “an” and “the” include pluralreferences unless the context clearly dictates otherwise. For example,the term “a compound” or “at least one compound” may include a pluralityof compounds, including mixtures thereof.

Throughout this application, various embodiments of this invention maybe presented in a range format. It should be understood that thedescription in range format is merely for convenience and brevity andshould not be construed as an inflexible limitation on the scope of theinvention. Accordingly, the description of a range should be consideredto have specifically disclosed all the possible subranges as well asindividual numerical values within that range. For example, descriptionof a range such as from 1 to 6 should be considered to have specificallydisclosed subranges such as the present example from 1 to 3, from 1 to4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well asindividual numbers within that range, for example, 1, 2, 3, 4, 5, and 6.This applies regardless of the breadth of the range.

Whenever a numerical range is indicated herein, it is meant to includeany cited numeral (fractional or integral) within the indicated range.The phrases “ranging/ranges between” a first indicate number and asecond indicate number and “ranging/ranges from” a first indicate number“to” a second indicate number are used herein interchangeably and aremeant to include the first and second indicated numbers and all thefractional and integral numerals therebetween.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination or as suitable in any other describedembodiment of the invention. Certain features described in the contextof various embodiments are not to be considered essential features ofthose embodiments, unless the embodiment is inoperative without thoseelements.

Various embodiments and aspects of the present invention as delineatedhereinabove and as claimed in the claims section below find experimentalsupport in the following examples.

EXAMPLES

Reference is now made to the following examples, which together with theabove descriptions illustrate some embodiments of the invention in a nonlimiting fashion.

The present examples present a technique for fusing the plurality ofchannels into a single color image with extended DOF and improvedappearance via a post-processing approach.

Out-of Focus Effects

An optical system can be described as a single “black box” element witha certain transfer function. For simplicity and without loss ofgenerality it is assumed in that the imaging system consists of a singlelens with a focal length f as shown FIG. 1. The object plane, lens planeand image plane coordinates are (ξ,η), (x,y) and (u,v) respectively,z_(o) is the distance from the object to the lens and z_(i) is thedistance from the lens to the image plane.

It is further assumed that the scenes are incoherently illuminated, sothat the optical system is linear with respect to the intensity [13]. Insuch case, the output image I_(i) can be expressed as the convolution ofthe imaging system point spread function (PSF) h₁ with the ideal imageintensity I_(g):I _(i)(u,v)=(h _(I) *I _(g))(u,v)  (EQ. 1.1)

The properties of such linear system can be analyzed in the spatialfrequency domain using Fourier transform representation. The normalizedFourier transform of the PSF is known as the Optical Transfer Function(OTF) and it describes the system response in terms of spatialfrequencies:

$\begin{matrix}{{{OTF}\left( {v_{x},v_{y}} \right)} = \frac{F\left\{ {h_{I}\left( {u,v} \right)} \right\}}{\int_{- \infty}^{\infty}{{h_{I}\left( {u,v} \right)}\ d\; u\; d\; v}}} & \left( {{EQ}.\mspace{14mu} 1.2} \right)\end{matrix}$

The OTF is the normalized autocorrelation of the system pupil functionP(x,y):OTF(v _(x) ,v _(y))=(P*P )(v _(x) ,v _(y))  (EQ. 1.3)

The Modulation transfer function (MTF) is the absolute value of the OTFand is one of the ways to describe imaging system properties. The MTFsubstantially provides the attenuation factor of all spatial frequenciespassing through the system.

The imaging condition defines the relation between z_(o), z_(i) and f,and can be written as:

$\begin{matrix}{{\frac{1}{z_{o}} + \frac{1}{z_{i}} - \frac{1}{f}} = 0} & \left( {{EQ}.\mspace{14mu} 1.4} \right)\end{matrix}$

When the imaging condition is satisfied, the PSF is proportional to theFourier transform of the pupil P. When imaging condition is notsatisfied, the imaging system suffers from out-of focus (OOF) aberrationwhich degrades the image quality (e.g., lower contrast level, loss ofsharpness and information).

In digital systems, when the image size of a point source in the objectplane is smaller than the pixel size in the detector plane, a smalldeviation from the imaging condition results in an OOF aberration thatis considered insignificant. The range in which the OOF aberration ofthe object is insignificant is known as the depth-of field (DOF).

The OOF aberration can be described as a wavefront error, e.g., a phaseerror in the pupil plane [13]. This phase error is represented as aquadratic phase term in the pupil plane. Thus, in the presence of OOFaberrations, the generalized pupil can be expressed as:

$\begin{matrix}{{\overset{\sim}{P}\left( {x,y} \right)} = {{P\left( {x,y} \right)} \cdot {\exp\left\lbrack {j\frac{\pi}{\lambda}\left( {\frac{1}{z_{o}} + \frac{1}{z_{img}} - \frac{1}{f}} \right)\left( {x^{2} + y^{2}} \right)} \right\rbrack}}} & \left( {{EQ}.\mspace{14mu} 1.5} \right)\end{matrix}$where z_(img) is the detector plane location when the object is in thenominal position, z_(o) is the actual object position and λ is theoptical wavelength. The special case in which the bracketed term isnull, is referred to as the in-focus condition.

In case of a circular aperture with radius R a defocus parameter Ψ isdefined as:

$\begin{matrix}{{\Psi = {{\frac{\pi\; R^{2}}{\lambda}\left( {\frac{1}{z_{o}} + \frac{1}{z_{img}} - \frac{1}{f}} \right)} = {\frac{\pi\; R^{2}}{\lambda}\left( {\frac{1}{z_{img}} - \frac{1}{z_{i}}} \right)}}},} & \left( {{EQ}.\mspace{14mu} 1.6} \right)\end{matrix}$and the generalized pupil in this case is:

$\begin{matrix}{{\overset{\sim}{P}\left( {x,y} \right)} = {{P\left( {x,y} \right)} \cdot {{\exp\left\lbrack {{j\Psi}\left( \frac{x^{2} + y^{2}}{R^{2}} \right)} \right\rbrack}.}}} & \left( {{EQ}.\mspace{14mu} 1.7} \right)\end{matrix}$

The defocus parameter Ψ describes the maximum phase error at theaperture edge. For Ψ>1 the image experiences contrast loss and for Ψ>4the image experiences information loss and even reversal of contrast forsome spatial frequencies, as shown in FIG. 2.

For a circular aperture, the diffraction limit maximum spatial frequency(also known as the cut-off frequency) is:

$\begin{matrix}{f_{c} = \frac{2R}{\lambda\; z_{i}}} & \left( {{EQ}.\mspace{14mu} 1.8} \right)\end{matrix}$

The resolution of the optical system rises as the aperture sizeincreases. At the same time, the DOF decreases as the defocus parameterΨ increases, per EQ. (1.6), thus reducing the resolution of OOF objects.

Optical Phase Mask

A binary optical phase mask with circular symmetry has been utilized inRef. [1]. This mask was composed of one or several rings providing apredetermined phase-shift (e.g., π). It was found by the presentinventors that such phase mask provides the exact desired phase shiftfor a single wavelength only, while the phase shift for other wavelengthchanges accordingly. In Ref. [2], an RGB phase mask was employed. Thisphase mask exhibits different responses in different color channels R, Gand B, as shown in FIGS. 3A-D and 4A-B. Each of the channels employed bythe RGB phase mask provides best performance for different fieldregions, so that the three channels jointly provide an extended DOF asshown in the simulation results exhibited by FIGS. 5A-R. The presentinventors successfully fused RGB channels into an improved color image.

Image Representation Using Overcomplete Dictionary

Natural images share common features over small patches. By using anovercomplete dictionary those patches can be presented as a linearcombination of a limited number of predefined patches from thedictionary.

Consider a signal column vector xϵR^(m) and a dictionary DϵR^(m×N)composed of N columns atoms signals. The signal x is said to have asparse approximation over D when there is a linear combination vectorαϵR^(N) of only a few atoms in D that provide an approximation of thesignal. This sparse approximation problem can be described as:

$\begin{matrix}{{\min\limits_{\alpha}{{x - {D\;\alpha}}}_{2}^{2}} + {\lambda{\alpha }_{0}}} & \left( {{EQ}.\mspace{14mu} 1.9} \right)\end{matrix}$

where λ is a regularization parameter.

The sparse approximation problem is known as NP-hard but can beefficiently solved with several known techniques including, withoutlimitation, a pursuit algorithm, e.g., Orthogonal Matching Pursuit(OMP), Matching Pursuit, Basis Pursuit, Order Recursive MatchingPursuit, Focal Underdetermined System Solver, or any combination orvariation thereof (see, for example, [14]). The dictionary D can beproduced from either predesigned transforms such as Wavelet, CTF and thelike, or from a set of sampled data from training images [3]. It wasfound by the present inventors that a training image can be used for thetraining data, and a classification process can be employed to create aglobal dictionary D composed of specially selected patches which can bereshaped to vector columns d_(i)ϵR^(64×1).

Non-Blind Image

A non-blind case, in which the blurring kernel is known, is consideredbefore the blind case.

When a sharp image S is blurred by a known kernel h, a blurred image Bis formed and can be described as:B=S*h+η  (EQ 1.10)Where η is additive Gaussian noise and * refers to the convolutionoperator. In the following, it is assumed that η<1. Using OMP thefollowing problem is solved

$\begin{matrix}{\alpha_{i} = {{\underset{\alpha}{\arg\;\min}{{b_{i} - {D_{b}\alpha_{i}}}}_{2}^{2}} + {\lambda{\alpha_{i}}_{0}}}} & \left( {{EQ}.\mspace{14mu} 1.11} \right)\end{matrix}$where α_(i) ϵR^(N) are the vector coefficients corresponding to the i-thpatch b_(i) ϵR^(m) from the blurred image and D_(b) ϵR^(m×N) is theblurred dictionary generated from the sharp dictionary D using the sameblurred kernel h. The solution of EQ. 1.11 produces the sparse codecoefficients α of the blurred image as a linear combination of atomsfrom the blurred dictionary D_(b). The sharp image can be recovered by Dand α. This process implies that for all i, D, α estimates of the sharppatch s_(i).

The relation between the two dictionaries allows restoring the originalimage from the blurry image. The restoration process can include adictionary learning process.

A known method for dictionary training is KSVD [15] which is aniterative process, alternating between finding the best vectorcoefficients using OMP and then updating D according to the currentcoefficients vector using Singular Value Decomposition (SVD).

In some embodiments of the present invention this approach is modifiedas follows. The blur kernel in most cases transfers only the low spatialfrequency in a specific patch while suppressing most of the highfrequencies as shown in FIGS. 6A-D. Thus, the difference between thefocused patches and the blurred patches is much higher for the randomchoice of patches (see FIGS. 6A and 6B) than that obtained with aselected set of patches consisting of primarily lower spatial frequencycomponents (FIGS. 6C and 6D). By choosing only low spatial frequenciespatches for the dictionary, the corresponding blurred patches resemblethe sharp patches and there is a better chance of selecting the correctsparse code coefficients α, thus making the dictionary more robust.

Blind Image

In the case of blind image deblurring the blur kernel is unknown and mayvary with the object position.

It was found by the present inventors that iterative processes such asthose described in [7-9] consume large computing resources. According tosome embodiments of the present invention an RGB optical phase mask isemployed so that the image is restored, optionally and preferablywithout any iterative process. In addition, the technique of the presentembodiments optionally and preferably handles both flat and depth scenesin the same manner, making it more robust for general real life scenes.

A natural scene can be described as a 3D scene consisting of a finitenumber of flat objects. Such a scene is referred to as a 2.5D scene.Each object in the scene is considered as being located in a singleplane. Without loss of generality, it is assumed that all objects in thescene are blurred with a blurring kernel that is affected only by thedefocus parameter Ψ of that scene. For this application one canconstruct a number of dictionaries created using a blurring kernel withdifferent defocus parameter. For example, when Δψ=1 is sufficientlydiscriminatory and Ψ spans from 0 to 6, it is sufficient to represent 7dictionaries (e.g., a first dictionary for ψ=0, a second dictionary forΨ=1, etc.). These dictionaries can be joined into one matrix, referredto herein as a “Multi-Focus Dictionary” (MFD). This procedure isschematically illustrated in FIG. 7.

In the present example, 8×8×3 patches or 192×1 vectors containing theRGB information for each patch were used to process color images. Thesharp in-focus dictionary D was composed of low-frequencies RGB vectors.The blurred MFD D_(b) was composed of a set of different dictionariesgenerated from D using different blurring kernels.

The blurry image can thus be described using D_(b). The OMP processchooses elements from the dictionary that best match the input patchbased on largest inner product. Using the RGB phase mask, the blurreddictionary varies strongly for different kernels since the response ofthe imaging system is very different for each color (see FIGS. 4A-B). Aninput patch (or vector) from an imaging system with the RGB mask alsoexhibits different response for each color. The responses are thereforedistinguishable among different kernels so that the likelihood that theinput vector is in accordance with vectors from D_(b) that experiencethe same blurring process is high. Thus, the technique of the presentembodiments improves the likelihood that the correct blurry elements areselected from D_(b).

Computer Simulation Results

The technique of the present embodiments was compared with the algorithmdisclosed by Krishnan, et al. [8]. The technique of the presentembodiments produced better results for natural images out of the Kodakdataset. The process was also employed on texture images (ColoredBrodatz Texture database) and observed similar performance. This resultdemonstrates that the MFD dictionary of the present embodiments can workon any natural scene.

In all cases the images were blurred by a blurring kernel with defocusparameter Ψ ranging from 0 to 6 with and without a phase mask. Blurredimages with mask were additionally restored using the method of thepresent embodiments.

FIGS. 8A-H are plots that show the results of the comparison. Dashedblue lines correspond to images blurred with phase mask, dashed greenlines correspond to images blurred without phase mask, solid blue andgreen lines correspond to restoration according to the teachings ofKrishnan, et al., and solid red lines correspond to restoration usingthe MFD of the present embodiments. Each plot refers to a differentimage and describes the peak signal-to-noise ratio (PSNR) as a functionof the defocus parameter. As shown in FIGS. 8A-H, the results obtainedaccording to the teachings of Krishnan, et al. presented lower value ofPSNR in all cases.

FIGS. 9A-F show example of image blurring and restoration, where FIG. 9Ashows the input image, FIGS. 9B and 9C show images corresponding toout-of-focus (Ψ=6) with clear aperture (FIG. 9B) and with phase mask(FIG. 9C), FIGS. 9D and 9E are results of de-blurring, respectivelyapplied to FIGS. 9B and 9C obtained according to the teachings ofKrishnan, et al, and FIG. 9F is a result of de-blurring applied to FIG.9C using the MFD of the present embodiments.

The technique of the present embodiments can also be applied to a 2.5Dscene, because the restoration that applies to one region of an imageframe is independent on the restoration process of other regions in thesame frame.

To demonstrate this process consider a 2.5D scene with four objects(three persons and the background) each located in a different distancefrom the camera as shown in FIG. 10A. The input image corresponding tothe 2.5D scene of FIG. 10A is shown in FIG. 10B. A depth map of the 2.5Dscene is shown in FIG. 10C, where the depths (values shown on the colorbar of FIG. 10C) are expressed in term of the defocus parameter Ψ.

In the present example, a pseudo-color representation is used for theidentification of different depth regions contained in the scene. Theprocedure according to some embodiments of the present invention isbased on the diversification due to acquisition of three different colorchannels in the presence of a chromatically dependent phase-mask (e.g.,an RGB phase mask).

FIGS. 11A-C show results obtained without RGB phase mask. In thisexample, the focus point was set to the background region of the scene(FIG. 11A). After restoration (FIG. 11B), the out-of-focus objects inthe front improved but the other object and the background are distortedand there are many visible artifacts. The depth map (FIG. 11C) showsthat most atoms were taken from the first sub-dictionary (Ψ=0) so thattherefore there was no distinction between different objects inside thescene.

FIGS. 12A-C show the results obtained with an imaging system comprisinga phase mask followed by the blind restoration image processing of thepresent embodiments. FIG. 12A shows the blurred image taken with a phasemask. FIG. 12B shows the restoration results. As shown, all objects arerestored without noticeable artifacts and with smooth transition betweenadjacent objects. FIG. 12C shows the depth map according to the mostlyused sub-dictionary in each area. As shown, in most areas, particularlythose with many details, the atoms were chosen mostly from thesub-dictionary that corresponds to the blurring kernel in that area.

Experimental Results

The technique of the present embodiments was tested on a table-top realobject scene using a commercial camera equipped with an RGB phase mask.The results are shown in FIGS. 13A-D, where FIGS. 13A and 13C showresults obtained without the phase mask and FIGS. 13B and 13D showresults obtained with the phase mask and the image processing techniquethat follows, according to the present embodiments.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims.

All publications, patents and patent applications mentioned in thisspecification are herein incorporated in their entirety by referenceinto the specification, to the same extent as if each individualpublication, patent or patent application was specifically andindividually indicated to be incorporated herein by reference. Inaddition, citation or identification of any reference in thisapplication shall not be construed as an admission that such referenceis available as prior art to the present invention. To the extent thatsection headings are used, they should not be construed as necessarilylimiting.

REFERENCES

-   [1] B. Milgrom, N. Konforti, M. A. Golub, and E. Marom, “Pupil    coding masks for imaging polychromatic scenes with high resolution    and extended depth of field,” Opt Express, vol. 18, pp. 15569-84,    Jul. 19, 2010.-   [2] B. Milgrom, N. Konforti, M. A. Golub, and E. Marom, “Novel    approach for extending the depth of field of Barcode decoders by    using RGB channels of information,” Opt Express, vol. 18, pp.    17027-39, Aug. 2, 2010.-   [3] M. Elad, Sparse and redundant representations: from theory to    applications in signal and image processing. New York: Springer,    2010.-   [4] F. Couzinie-Devy, J. Mairal, F. Bach, and J. Ponce, “Dictionary    learning for deblurring and digital zoom,” arXiv preprint    arXiv:1110.0957, 2011.-   [5] H. Huanga and N. Xiaoa, “Image Deblurring Based on Sparse Model    with Dictionary Learning,” 2013.-   [6] Q. Shan, J. Jia, and A. Agarwala, “High-quality motion    deblurring from a single image,” in ACM Transactions on Graphics    (TOG), 2008, p. 73.-   [7] Z. Hu, J. B. Huang, and M. H. Yang, “Single image deblurring    with adaptive dictionary learning,” in Image Processing (ICIP), 2010    17th IEEE International Conference on, 2010, pp. 1169-1172.-   [8] D. Krishnan, T. Tay, and R. Fergus, “Blind deconvolution using a    normalized sparsity measure,” in Computer Vision and Pattern    Recognition (CVPR), 2011 IEEE Conference on, 2011, pp. 233-240.-   [9] H. Zhang, J. Yang, Y. Zhang, and T. S. Huang, “Sparse    representation based blind image deblurring,” in Multimedia and Expo    (ICME), 2011 IEEE International Conference on, 2011, pp. 1-6.-   [10] R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P.    Hanrahan, “Light field photography with a hand-held plenoptic    camera,” Computer Science Technical Report CSTR, vol. 2, 2005.-   [11] A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and    depth from a conventional camera with a coded aperture,” ACM    Transactions on Graphics (TOG), vol. 26, p. 70, 2007.-   [12] F. Guichard, H.-P. Nguyen, R. Tessières, M. Pyanet, I.    Tarchouna, and F. Cao, “Extended depth-of-field using sharpness    transport across color channels,” in IS &T/SPIE Electronic Imaging,    2009, pp. 72500N-72500N-12.-   [13] J. W. Goodman, Introduction to Fourier optics, 2nd ed. New    York: McGraw-Hill, 1996.-   [14] S. S. Chen, D. L. Donoho, and M. A. Saunders, “Atomic    decomposition by basis pursuit,” SIAM journal on scientific    computing, vol. 20, pp. 33-61, 1998.-   [15] M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for    designing overcomplete dictionaries for sparse representation,” Ieee    Transactions on Signal Processing, vol. 54, pp. 4311-4322, November    2006.

What is claimed is:
 1. A method of processing an image, comprising:decomposing the image into a plurality of channels, each beingcharacterized by a different depth-of-field, and storing said decomposedimage in a memory; accessing, by a digital image processor, anon-transitory computer readable medium storing an in-focus dictionarydefined over a plurality of dictionary atoms, and an out-of-focusdictionary defined over a plurality of sets of dictionary atoms, eachset corresponding to a different out-of-focus condition; computing, bysaid image processor, at least one sparse representation of saiddecomposed image over said dictionaries, and storing said at least onesparse representation in a memory, wherein said image is divided into aplurality of patches, and wherein said computing is performed byexpressing each patch as a combination of atoms of a sub-dictionary ofsaid out-of-focus dictionary; and generating by said image processor aprocessed image using said sparse representation, and displaying saidprocessed image on a display device.
 2. The method of claim 1, whereinsaid decomposing is done optically at the time of image capture.
 3. Themethod of claim 1, wherein said decomposing is done by digital imageprocessing.
 4. The method according to claim 1, wherein said sparserepresentation is computed such that different region of said image havedifferent defocus conditions.
 5. A method of imaging, comprising:capturing an image of a scene using an imaging device having an opticalmask selected to optically decompose said image into a plurality ofchannels, each being characterized by a different depth-dependence of aspatial frequency response of said imaging device; by a digital imageprocessor, accessing a non-transitory computer readable medium storingan in-focus dictionary defined over a plurality of dictionary atoms, andan out-of-focus dictionary defined over a plurality of sets ofdictionary atoms, each set corresponding to a different out-of-focuscondition; computing, by said image processor, at least one sparserepresentation of said decomposed image over said dictionaries, andstoring said at least one sparse representation in a memory, whereinsaid image is divided into a plurality of patches, and wherein saidcomputing is performed by expressing each patch as a combination ofatoms of a sub-dictionary of said out-of-focus dictionary and generatingby said image processor a processed image using said sparserepresentation, and displaying said processed image on a display device.6. The method according to claim 1, further comprising constructing adepth map of the scene, based on said sparse representation.
 7. Themethod according to claim 6, further comprising re-focusing at least aportion of said image based on said depth map.
 8. The method accordingto claim 6, further comprising generating a three-dimensional imagebased on said depth map.
 9. The method according to claim 6, whereinsaid depth map is constructed based on a single image frame.
 10. Animaging system, comprise: an imaging device having an optical mask andbeing configured for capturing an image of a scene, wherein said opticalmask is selected to optically decompose said image into a plurality ofchannels, each being characterized by a different depth-dependence of aspatial frequency response of said imaging device; a non-transitorycomputer readable medium storing an in-focus dictionary defined over aplurality of dictionary atoms, and an out-of-focus dictionary definedover a plurality of sets of dictionary atoms, each set corresponding toa different out-of-focus condition; and a digital image processorconfigured for accessing said computer readable medium, for computingand storing in a memory at least one sparse representation of saiddecomposed image over said dictionaries, for generating a processedimage using said sparse representation, and for displaying saidprocessed image on a display device, wherein said image is divided intoa plurality of patches, and wherein said computing is performed byexpressing each patch as a combination of atoms of a sub-dictionary ofsaid out-of-focus dictionary.
 11. A portable device, comprising theimaging system of claim
 10. 12. The portable device of claim 11, beingselected from the group consisting of a cellular phone, a smartphone, atablet device, a mobile digital camera, a wearable camera, a personalcomputer, a laptop, a portable media player, a portable gaming device, aportable digital assistant device, and a portable navigation device. 13.The imaging system according to claim 10, wherein said data processor isconfigured for constructing a depth map of the scene, based on saidsparse representation.
 14. The imaging system according to claim 13,wherein said data processor is configured for re-focusing at least aportion of said image based on said depth map.
 15. The imaging systemaccording to claim 13, wherein said data processor is configured forgenerating a three-dimensional image based on said depth map.
 16. Theimaging system according to claim 13, wherein said depth map isconstructed based on a single image frame.
 17. A computer softwareproduct, comprising a non-transitory computer-readable medium in whichan in-focus dictionary, an out-of-focus dictionary and programinstructions are stored, wherein said in-focus dictionary is definedover a plurality of dictionary atoms, and said out-of-focus dictionaryis defined over a plurality of sets of dictionary atoms, each setcorresponding to a different out-of-focus condition, and wherein saidinstructions, when read by digital image processor, cause the imageprocessor to receive an image, to decompose the image into a pluralityof channels and to store said decomposed image in a memory, to accesssaid in-focus dictionary and said out-of-focus dictionary, to computeand store in a memory at least one sparse representation of saiddecomposed image over said dictionaries to generate a processed imageusing said sparse representation, and to display said processed image ona display device, wherein said image is divided into a plurality ofpatches, and wherein said computing is performed by expressing eachpatch as a combination of atoms of a sub-dictionary of said out-of-focusdictionary.
 18. The computer software product of claim 17, wherein saidsparse representation is computed such that different region of saidimage have different defocus conditions.
 19. The method according toclaim 1, wherein said plurality of channels is a plurality of colorchannels.
 20. The method according to claim 1, wherein said computingsaid sparse representation is executed without iteration.
 21. The methodaccording to claim 1, wherein each patch of at least a few of saidpatches overlaps with at least one adjacent patch.
 22. The methodaccording to claim 1, wherein said few patches comprise 50% of saidpatches.
 23. The method according to claim 1, wherein for each of atleast a few pairs of adjacent patches, said overlap equals at least 50%of an area of each patch of said pair.
 24. The method according to claim1, wherein said out-of-focus dictionary comprises a plurality ofsub-dictionaries, each being characterized by a defocus parameter, andwherein different sub-dictionaries correspond to different values ofsaid defocus parameter.
 25. The method according to claim 24, whereineach of said plurality of sub-dictionaries is obtainable from saidin-focus dictionary by an inner product of said in-focus dictionary by akernel function characterized by a respective defocus parameter.
 26. Themethod according to claim 24, wherein said out-of-focus dictionarycomprises at least three sub-dictionaries.
 27. The method according toclaim 24, wherein said out-of-focus dictionary comprises at least foursub-dictionaries.
 28. The method according to claim 24, wherein saidout-of-focus dictionary comprises at least five sub-dictionaries. 29.The method according to claim 24, wherein said out-of-focus dictionarycomprises at least six sub-dictionaries.
 30. The method according toclaim 24, wherein said out-of-focus dictionary comprises at least sevensub-dictionaries.
 31. The method according to claim 1, wherein at leastP % of atoms of said in-focus dictionary are characterized by a spatialfrequency which is at most a predetermined cutoff frequency, saidpredetermined cutoff frequency corresponding to at most T transitionsbetween dark and bright picture-elements along any straight line acrossa respective atom, wherein P is at least 50, wherein T equals └0.5 L┘,and wherein L is a width of said atom.
 32. A method of processing animage, comprising: decomposing the image into a plurality of channels,each being characterized by a different depth-of-field, and storing saiddecomposed image in a memory; accessing, by a digital image processor, anon-transitory computer readable medium storing an in-focus dictionarydefined over a plurality of dictionary atoms, and an out-of-focusdictionary defined over a plurality of sets of dictionary atoms, eachset corresponding to a different out-of-focus condition, wherein saidout-of-focus dictionary comprises a plurality of sub-dictionaries, eachbeing characterized by a defocus parameter, and wherein differentsub-dictionaries correspond to different values of said defocusparameter; computing, by said image processor, at least one sparserepresentation of said decomposed image over said dictionaries andstoring said at least one sparse representation in a memory; andgenerating by said image processor a processed image using said sparserepresentation, and displaying said processed image on a display device.33. The method of claim 32, wherein said decomposing is done opticallyat the time of image capture.
 34. The method of claim 32, wherein saiddecomposing is done by digital image processing.
 35. The methodaccording to claim 32, wherein said sparse representation is computedsuch that different region of said image have different defocusconditions.
 36. The method according to claim 32, further comprisingconstructing a depth map of the scene, based on said sparserepresentation.
 37. The method according to claim 36, further comprisingre-focusing at least a portion of said image based on said depth map.38. The method according to claim 36, further comprising generating athree-dimensional image based on said depth map.
 39. The methodaccording to claim 36, wherein said depth map is constructed based on asingle image frame.
 40. An imaging system, comprise: an imaging devicehaving an optical mask and being configured for capturing an image of ascene, wherein said optical mask is selected to optically decompose saidimage into a plurality of channels, each being characterized by adifferent depth-dependence of a spatial frequency response of saidimaging device; a non-transitory computer readable medium storing anin-focus dictionary defined over a plurality of dictionary atoms, and anout-of-focus dictionary defined over a plurality of sets of dictionaryatoms, each set corresponding to a different out-of-focus condition,wherein said out-of-focus dictionary comprises a plurality ofsub-dictionaries, each being characterized by a defocus parameter, andwherein different sub-dictionaries correspond to different values ofsaid defocus parameter; and a digital image processor configured foraccessing said computer readable medium, for computing and storing in amemory at least one sparse representation of said decomposed image oversaid dictionaries, for generating a processed image using said sparserepresentation, and for displaying said processed image on a displaydevice.
 41. A portable device, comprising the imaging system of claim10.
 42. The portable device of claim 41, being selected from the groupconsisting of a cellular phone, a smartphone, a tablet device, a mobiledigital camera, a wearable camera, a personal computer, a laptop, aportable media player, a portable gaming device, a portable digitalassistant device, and a portable navigation device.
 43. The imagingsystem according to claim 40, wherein said data processor is configuredfor constructing a depth map of the scene, based on said sparserepresentation.
 44. The imaging system according to claim 43, whereinsaid data processor is configured for re-focusing at least a portion ofsaid image based on said depth map.
 45. The imaging system according toclaim 43, wherein said data processor is configured for generating athree-dimensional image based on said depth map.
 46. The imaging systemaccording to claim 43, wherein said depth map is constructed based on asingle image frame.
 47. The imaging system according to claim 40,wherein each of said plurality of sub-dictionaries is obtainable fromsaid in-focus dictionary by an inner product of said in-focus dictionaryby a kernel function characterized by a respective defocus parameter.48. The imaging system according to claim 40, wherein a number ofsub-dictionaries in said out-of-focus dictionary is selected from thegroup consisting of at least three sub-dictionaries, at least foursub-dictionaries, at least five sub-dictionaries, at least sixsub-dictionaries and at least seven sub-dictionaries.
 49. A method ofprocessing an image, comprising: decomposing the image into a pluralityof channels, each being characterized by a different depth-of-field, andstoring said decomposed image in a memory; accessing, by a digital imageprocessor, a non-transitory computer readable medium storing an in-focusdictionary defined over a plurality of dictionary atoms, and anout-of-focus dictionary defined over a plurality of sets of dictionaryatoms, each set corresponding to a different out-of-focus condition,wherein at least P % of atoms of said in-focus dictionary arecharacterized by a spatial frequency which is at most a predeterminedcutoff frequency, said predetermined cutoff frequency corresponding toat most T transitions between dark and bright picture-elements along anystraight line across a respective atom, wherein P is at least 50,wherein T equals └0.5 L┘, and wherein L is a width of said atom;computing, by said image processor, at least one sparse representationof said decomposed image over said dictionaries and storing said atleast one sparse representation in a memory; and generating by saidimage processor a processed image using said sparse representation, anddisplaying said processed image on a display device.